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1. Introduction 

A strong impetus for this paper came, at least for the first author, from a 
question of Avner Ash, asking whether one can construct non-selfdual, non- 
monomial cuspidal cohomology classes for suitable congruence subgroups T 
of SL(n,Z), say for n = 6. Such a construction, in special examples, has 
been know n for some time for n = 3 f |AGG1984| . |vGT1994j . |vGKTV1997j . 
|vGT2000] ): it is of course not possible for n = 2. One can without trou- 
ble construct non-selfdual, monomial classes for any n = 2m with m > 2, 
not just for constant coefficients (see the Appendix, Theorem E). In the 
Appendix we also construct non-monomial, non-selfdual classes for n = 4 
using the automorphic induction to Q of suitable Hecke character twists of 
non-CM cusp forms of "weight 2" over imaginary quadratic fields, but they 
admit quadratic self-twists and are hence imprimitive. The tack pursued 
in the main body of this paper, and which is the natural thing to do, is to 
take a non-selfdual (non-monomial) n = 3 example tt, and take its functo- 
rial product M with a cuspidal tt' on GL(2)/Q associated to a holomorphic 
newform of weight 4 for a congruence subgroup of SL(2,Z). The resulting 
(cohomological) n = 6 example can be shown to be non-selfdual for suitable 
tt' . (This should be the case for all tt' , but we cannot prove this with current 
technology - see Remark 4.1.) Given that, the main problem is that it is 
not easy to show that such an automorphic tensor product H. := ttM tt' , 
whose modularity was established in the recent deep work of H. Kim and 
F. Shahidi ( KSh2002-lJ, is cuspidal. This has led us to prove a precise 
cuspidality criterion (Theorem A) for this product, not just for those of co- 
homological type, which hopefully justifies the existence of this paper. The 
second author earlier proved such a criterion when tt is a twist of the sym- 
metric square of a cusp form on GL(2) ( Wa2()()3 : such forms are essentially 
selfdual, however, and so do not help towards the problem of constructing 
non-selfdual classes. One of the reasons we are able to prove the criterion in 
general is the fact that the associated, degree 20 exterior cube L-function is 
nicely behaved and analyzable. This helps us rule out, when the forms on 
GL(2) and GL(3) are non-monomial, the possible decomposition of II into 
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an isobaric sum of two cusp forms on GL(3) (see section 7). This is the 
heart of the matter. 

We will also give a criterion (Theorem B) as to when the base change of 
II to a solvable Galois extension remains cuspidal. We will derive a stronger 
result for the cohomological examples (Theorem C), namely that each of 
them is primitive, i.e., not associated to a cusp form on GL(m)/K for any 
(possibly non- normal) extension K/Q of degree d > 1 with dm = 6. Further- 
more, each of the three main non- selfdual GL(3) examples tt of |vGT1994j . 
|vOKTVT997| and jvGT2000 j comes equipped, confirming a basic conjec- 
ture of Clozel f |C1988| L with a certain 3-dimensional £-&dic representation 
Pl whose Frobenius traces a p (pi) agree with the Hecke eigenvalues a p (ir) for 
small p. For tt' on GL(2)/Q defined by a suitable holomorphic newform of 
weight 4, with associated Galois representation p' e , we will show (Theorem 
D) that the six-dimensional Kg := pi® p'^, which should correspond to II, 
remains irreducible under restriction to any open subgroup of Gal(Q/Q). 

The first author would like to thank Avner Ash for his question and for 
making comments on a earlier version of this paper, Mahdi Asgari for ini- 
tially kindling his interest (at Park City, UT) in the problem of establishing 
a precise cuspidality criterion for the Kim-Shahidi product, and the National 
Science Foundation for financial support through the grant DMS-0100372. 
The second author would like to thank James Cogdell and Henry Kim for 
their interest in his lecture on this work at the Fields Institute Workshop 
on Automorphic L-functions in May 2003. 

2. The Cuspidality Criterion 

Throughout this paper, by a cusp form on GL(n) (over a global field F) we 
will mean an irreducible, cuspidal automorphic representation tt = ir^ <g> 717 
of GL u (Af). We will denote its central character by av. One says that 
tt is essentially self-dual iff its contragredient tt v is isomorphic to tt <g> v 
for some character v of (the idele classes of) F; when n = 2, one always 
has 7r v ~ tt ® lo^ 1 - Note that tt is unitary iff tt v is the complex conjugate 
representation W. Given any cusp form tt, we can find a real number t such 
that tt u := tt <8> |.|* is unitary. 

For any cusp form tt' on GL{2), put Ad{Tx') = sym 2 (7r / )C3a'~, 1 and A a (tt') = 
sym 4 (-7r') (g) w~/ 2 . Recall that tt' is dihedral iff it admits a self- twist by a 
quadratic character; it is tetrahedral, resp. octahedral, iff sym 2 (Tr'), resp. 
sym 3 (Tr'), is cuspidal and admits a self-twist by a cubic, resp. quadratic, 
character. (The automorphy of sym 5 (Tr') was shown by Kim and Shahidi 
in |KSh2002-lj .) We will say that tt' is of solvable polyhedral type iff it is 
dihedral, tetrahedral or octahedral. 

Theorem A. Let tt',tt be cusp forms on GL{2), GL(3) respectively over a 
number field F. Then the Kim-Shahidi transfer II = tt M tt' on GL(6)/F is 
cuspidal unless one of the following happens: 
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(a) tt' is not dihedral, and tt is a twist of Ad(ir'); 

(b) tt' is dihedral, L(s, tt) = L(s, x) for an idele class character x of a cu- 
bic, non-normal extension K of F, and the base change ttk is Eisensteinian. 

Furthermore, when (a) (resp. (b)) happens, H possesses an isobaric de- 
composition of type (2,4) or (2,2,2) (resp. of type (3, 3)). More precisely, 
when we are in case (a), II is of type (2,2,2) if tt' is tetrahedral, and (2,4) 
otherwise. 

Remark: By |KSh2002-lj . II = tt H tt' is automorphic on GL(6)/F, 
and its L-function agrees with the Rankin-Selberg L-function L(s,tt x tt'). 
Theorem A implies in particular that II is cuspidal if (i) tt' is not dihedral 
and (ii) tt is not a twist of Ad(TT ! ). 

A partial cuspidality criterion was proved earlier by the second author in 
Wa2003 ; but he only treated the case when tt is twist equivalent to the 
Gelbart-Jacquet symmetric square transfer of some cusp form on GL(2). 

Theorem B. Let F be a number field and tt' , tt be cusp forms on GL(2)/F, 
GL(3)/F respectively. Put LT = tt Kl tt' . Assume that tt' is not of solvable 
polyhedral type, and tt not essentially selfdual. Then we have the following: 

(a) If tt does not admit any self twist, II is cuspidal without any self 
twist. Furthermore, if tt is not monomial, then LT is not induced from any 
non-normal cubic extension. 

(b) If tt is not of solvable type, i.e., its base change to any solvable Galois 
extension is cuspidal, II is cuspidal and not of solvable type; in particular, 
there is no solvable extension K/F of degree d > 1 dividing 6, and a cuspidal 
automorphic representation rj of GL e / d (Ap), such that L(s,II) = L(s,i]). 

Remark: If tt is regular algebraic at infinity, and F is not totally imagi- 
nary, then tt is not monomial (See Lemma l9.3|) . 

We will prove Theorem EI in sections 6 through 8, and Theorem El in 
section 9. 

Before proceeding with the proofs of these theorems, we will digress and 
discuss the cohomological application. 

3. Preliminaries on cuspidal cohomology 

The experts can skip this section and go straight to the statement (in 
section 4) and the proof (in section 5) of Theorems \d\ iDl Let 

T c SL(n,Z), 

be a congruence subgroup of := SL(n,M), which has finite covolume. T 
acts by left translation on the symmetric space X® := SL(n,R)/SO(n). The 
cohomology of F is the same as that of the locally symmetric orbifold T\A^. 
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If H* ont denotes the continuous group cohomology, a version of Shapiro's 
lemma gives an isomorphism 

H*(T,C) ~ i?* ont (G°,C°°(r\G°)). 

The constant functions are in this space, and the contribution of H* ont (G^, C) 
to H*(T,C) is well understood and plays an important role in Borel's in- 
terpretation of the values of the Riemann zeta function £(s) at negative 
integers. 

We will be interested here in another, more mysterious, piece of H*(T, C), 
namely its cuspidal part, denoted H* nsp (T,C), which injects into H*(T, C) 
by a theorem of Borel. Furthermore, one knows by L. Clozel (101988!) that 
the cuspidal summand is defined over Q, preserved by the Hecke operators. 
The cuspidal cohomology is represented by cocycles defined by smooth cusp 
forms in L 2 (r\G*), i.e., one has 

-^cus P (r,C) = H* ont (Gn , L 2 usp (r\G°)°°), 

where L 2 (T\G^) denotes the space of cusp forms, and the superscript 
oo signifies taking the subspace of smooth vectors. If (5 n denotes the com- 
plexified Lie algebra of G n , the passage from continuous cohomology to the 
relative Lie algebra cohomology ( BoW1980 ) furnishes an isomorphism 

i?: usp (r,C) ~ H*(& n ,K:L 2 cusp (T\G° n r). 

It is a standard fact (see BoJ1979], for example) that the right action of G n 
on L^ usp (r\G°) is completely reducible, and so we may write 

£cus P (r\Gn) ~ ^rr^TU, 

where ir runs over the irreducible unitary representations of (up to equiv- 
alence), Ti. n denotes the space of ir, © signifies taking the Hilbert direct sum, 
and m n is the multiplicity. Consequently, 

One knows completely which representations 7r of G° have non-zero (<5„, K)- 
cohomology QVZ1984] : see also |Kul98 _). An immediate consequence (see 
C1988_, page 114) is the following (with [x] denoting, for any x £ R, the 
integral part of x): 

Theorem 3.1. 

H l cusp (T,C) = unless d{n) < i < d(n) + [(n - l)/2], 

where 

d(n) = m 2 if n = 2m and d(n) = m(m + 1) if n = 2m + 1. 

It will be necessary for us to work with the Q-group G n := GL(n) with 
center Z n , and also work adelically. Let A = R x Af be the adele ring of Q, 
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Kqo = 0(n), and X n = G n (R) / K n , whose connected component is X®. For 
any compact open subgroup K of G n (Af), we have 

(3.6) S K := G n (Q)Z n (R) \G n (A)/K oo K ~ if^T^X*, 

where the Tj are congruence subgroups of SL(n,Z) and Z n (R)° is the Eu- 
clidean connected component of Z n (R). We need the following, which follows 
easily from the discussion in section 3.5 of |Cl988 : 

Theorem 3.2. (i) 

^cusp(^)C) ~ ©7reCoh K H*(<3 nt00 , Koo; Tr^) (g) irf, 

where njO o consists of matrices in M n (C) with purely imaginary 
trace, and Cohfr- is the set of (equivalence classes) of cuspidal auto- 
morphic representations tt = tt^ £g> ttj of G n (A) such that tt^ ^ 0, 
iToo contributes to the relative Lie algebra cohomology, and (uj^)^ is 
trivial on Z(M.)°. 

(ii) Suppose tt = ir^ <g> nt is a cuspidal automorphic representation of 
G n (A) with irf ^ such that the restriction r^ of the Langlands 
parameter of tToq to C* is given by the n-tuple 

{(z/\z\r~\(z/\z\r-\(z/\z\r- 3 ,(z/\z\r- 3 , 
...^z/wuz/m^izzY- 1 

if n is even, and 

{(z/\z\T-\(z/\z\Y-\(z/\z\Y-\(z/\z\Y-\ 
...,{z/\z\f,{z/\z\)\l}®{zzT- 1 
if n is odd. Then tt contributes to Coh^ in degree d(n) . 

□ 

Given any cohomological tt as above, the fact that the cuspidal cohomol- 
ogy at any level K has a Q-structure (tC1988j) preserved by the action of 
the Hecke algebra Ti.Q(Gf,K) (consisting of Q-linear combinations of K- 
double cosets), implies that the G/-module ttj is rational over a number 
field Q(vrj). When n = 2, such a tt is defined by a holomorphic newform h 
of weight 2, and then Q(7Tj) is none other than the field generated by the 
Fourier coefficients of h. 

4. NON-SELFDUAL, CUSPIDAL CLASSES FOR T C SL(6, Z) 

The principle of functoriality predicts that given cuspidal automorphic 
representations tt,tt' of G n (A),G m (A) respectively, there exists an isobaric 
automorphic representation tt M tt' of G nm (A) such that for every place v of 
Q, one has 

cj{{tt M tt') v ) ~ o{tt v ) <g> u v {tt' v ), 
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where a is the map (up to isomorphism) given by the local Langlands cor- 
respondence (fHaT2000 , He2000 ) from admissible irreducible representa- 
tions of G r (Q„) to r-dimensional representations of W^, which is the real 
Weil group Wr if v = oo and the extended Weil group Wq x SL(2, C) if v 
is defined by a prime number p. 

This prediction is known to be true for n = m = 2 ( Ra2000 ), More 
importantly for the matter at hand, it is also known for (n, m) = (3, 2) by 
a difficult theorem of H. Kim and F. Shahidi ( |KSh2002-l] ). 

Put 

T = Ti UT 2 , 

with 

Ti = {53, 61, 79, 89} and T 2 = {128, 160, 205}. 

By the article AGG1984 of Ash, Grayson and Green (for p G Ti), and the 
works |vGT1994j . |vGT2000j . |vGKTVT997| of B. van Geemen, J. Top, et 
al (for p G T 2 ), one knows that for every q G T, there is a non-selfdual cusp 
form 7r(q) on GL(3)/Q of level q, contributing to the (cuspidal) cohomology 
(with constant coefficients) . 

Theorem C. Let tt' be a cusp form on GL(2)/Q defined by a non-CM 
holomorphic newform g of weight A, level N , trivial character, and field of 
coefficients K . Letn denote an arbitrary cusp form on GL(3)/Q) contributing 
to the cuspidal cohomology in degree 2, and let Tr(q), q G T , be one of the 
particular forms discussed above. Put IT = tt M it' and 11(g) = 7r(q) Kl it' . 
Then 

(a) IT contributes to the cuspidal cohomology of GL(6). 

(b) n(g) is not essentially self dual when N < 23 and K = Q. 

(c) If N is relatively prime to q, then the level of H(q) is N 3 q 2 . Now 
let N < 23 and K = Q. Then H(q) does not admit any self-twist. 
Moreover, there is no cubic non-normal extension K/Q with a cusp 
form rj on GL(2)/K such that L(s,Tl(q)) = L(s,r]), nor is there a 
sextic extension (normal or not) E/Q with a character A of E such 
that L(s,U(q)) = L(s, A). 

We note from the Modular forms database of William Stein ( WSt2003 ) 
that there exist newforms g of weight 4 with Q-coefficients, for instance for 
the levels JV = 5, 7, 13, 17, 19, 23. 

Remark 4-1 Part (b) should be true for any tt' . Suppose that for a given 
any cusp form tt on GL(3), cohomological or not, the functorial product 
IT = tt M tt' satisfies n v ~ IT (g) v for a character v. Then at any prime p 
where a p {Tr') 7^ 0, which happens for a set of density 1, we can of course 
conclude that a p (7r v ) = a p (-TT)i'(p). But this does not suffice, given the state 
of knowledge right now concerning the refinement of the strong multiplicity 
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one theorem, to conclude that isomorphic to a twist of tt. In the case of 
the 7r(g), we have information at a small set of primes and we have to make 
sure that a p (Tr') ^ and a p (7r) ^ a p (ir) for one of those p. The hypothesis 
that K = Q is made for convenience, however, and the proof will extend to 
any totally real field. 

In |v(;T1994| . |v(;T20n0j . |vGKTV1997| one finds in fact an algebraic 
surface S{q) over Q for each q E T2, and a 3-dimensional £-adic representa- 
tion p(q) (for any prime £), occurring in H^ t (S (q)^, Qi) , such that 

(*) L p (s,p{q)) = L p (s,TT(q)), 

for all odd primes p < 173 not dividing q. Here is a conditional result. 

Theorem D. Let tt' be a cusp form on GL(2)/Q defined by a non-CM 
holomorphic newform g of weight 4, level N , trivial character, and field of 
coefficients Q, with corresponding ^-representation p' . Let Tr(q), p(q),U(q) 
be as above for q £ T2. Put R(q) = p(q) (g) p' ■ Suppose (*) holds at all the 
odd primes p not dividing ql. Then R(q) remains irreducible when restricted 
to any open subgroup of Ga/(Q/Q). 



5. Proof of Theorems C, D modulo Theorems A, B 

In this section we will assume the truth of Theorems and IB1 
Proof of Theorem As tt' is associated to a holomorphic newform of 
weight 4, we have 

'(Olc * {(z/\z\f®(z/\z\f)®(zzf. 

And since tt contributes to cohomology, we have (cf. part (ii) of Theorem l3.1j) 

^(7Too)b - {{z/\z\f @1@{Z/\Z\) 2 )®{ZZ) 2 . 

Since Iloo corresponds to the tensor product u (71-00) (g) 17(71"^), we get part 
(a) of Theorem |U] in view of Theorem [X] and part (ii) of Theorem 13.21 

Pick any q in T and denote by Q(7r((/)) the field of rationality of the finite 
part Tr{q)f of Tr{q). Then it is known by [AGG1984J that for q £ T\, 

Q(tt(53)) = Q(v /r lT), Q(vr(61)) = Q(V^S), Q(tt(79)) = Q(\/ z 15), Q(tt(89)) 
while by |vGT1994|. |vGKTV1997| and ^SHM, 

Q(tt(?)) = Q(i), V q G T 2 . 

By hypothesis, tt' is non-CM, and by part (a), H(q) is cuspidal. Suppose 
there exists a character v such that for some q £ T, 

n(g) v ~ 11(g) (8) v. 
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Comparing central characters, we get u e = 1. We claim that is 2 = 1. Sup- 
pose not. Then there exists an element a of Gal(Q/Q) fixing Q(7r(g)) such 
that v ^ u a . Since tt' has Q-coefficients and ir(q) has coefficients in Q)(ir(q)), 
we see that n(q)j must be isomorphic to the Galois conjugate H(q)f, which 
exists because the cuspidal cohomology group has, by Clozel (see section 3), 
a Q-structure preserved by the Hecke operators. If we put /i = v/v" ^ 1, 
we then see that H(q) ~ n(g) (g) /x. But we will see below that I\.{q) admits 
no non-trivial self-twist. This gives the desired contradiction, proving the 
claim. If v is non-trivial, the quadratic extension F/Q it cuts out will need 
to have discriminant dividing q a N b for suitable integers a, b. For any prime 
p which is unramified in F, we will have 

a p (ir)a p (-K') = ±a p (iT)a p (iT'). 

For each j < 3 and for each tt' with N < 23 and K = Q, we can find, 
using the tables in |AGG1984j . |vGT1994| . jvGT2000j . |vGKTV1997j and 
WSt2003 , a prime p such that a p (7r') / 0, v(p) ^ and a p {ir) ^ a p (ir). 
This proves part (b) of Theorem ICl 

When N is relatively prime to q, the conductor of 11(g) must be iV 3 g 2 as 
can be seen from the way epsilon factors change under twisting (see section 
4 of |BaR1994j for example). 

From now on, let N < 23 and K = Q. One knows that as tt' is holo- 
morphic and not dihedral, the associated Galois representation p' remains 
irreducible when restricted to any open subgroup of Gal(Q/Q). It follows 
that the base change of %' to any solvable Galois extension remains cuspidal. 
In particular, it is not of solvable polyhedral type. We claim that 7r((?) is not 
monomial. Indeed, the infinite type of ir(q) is regular algebraic C1988], and 
to be monomial there needs to be a cubic, possibly non-normal, extension 
K/Q which can support an algebraic Hecke character which is not a finite 
order character times a power of the norm. By |Wel955j . for such a charac- 
ter to exist, K must contain a CM field, i.e., a totally imaginary quadratic 
extension of a totally real field, which forces K to be imaginary. But any 
cubic extension of Q has a real embedding, and this proves our claim. Note 
also that as n{q) is not essentially self-dual, it is not a twist of the symmetric 
square of any cusp form, in particular tt', on GL(2)/Q. Now it follows from 
Theorem El that H(q) does not admit any self-twist. 

Suppose if is a non-normal cubic field together with a cusp form rj on 
GL(2)/K such that L(s,H(q)) = L(s,rj). Let L be the Galois closure of K 
(with Galois group S3), and let E be the quadratic extension of Q contained 
in L. Then H(q)E will be cuspidal and automorphically induced by the cusp 
form rji of GL(3, A^). In other words, H(q)E admits a non-trivial self-twist. 
To contradict this, it suffices, in view of Theorem El to show that vt((j)e 
admits no self-twist relative to L/E, i.e., that ix{q)E is not automorphically 
induced by a character /i of L. But as noted above, this forces L to be 
a totally imaginary number field containing a CM field Lq. Then either 
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L = Lq or Lq = E. In the latter case, by jWe!955j . p, will be a finite order 
character times the pullback by norm of a character of E, forcing I^{fi) 
to be not regular at infinity, and so this case cannot happen. So L itself 
must be a CM field, with its totally real subfield F. Then Ga\(F/Q) would 
be cyclic of order 3 and a quotient of S3, which is impossible. So this case 
does not arise either. So ft{q)E doe snot admit any self-twist, and 11(g) is 
not associated to any r/ as above. 

Now suppose L(s,IL(q)) = L(s, A) for a character A of a sextic field L. If 
L contains a proper subfield M/Q, then since m := [L : M] < 3, one can 
induce A to M and get an automorphic representation of GL m (&M) such 
that L(s, A) = L(s,(3) = L(s, 11(g)), which is impossible by what we have 
seen above. So L must not contain any such M. But on the other hand, 
since n(g)oo is algebraic and regular, we need L to contain, by |Wel955j . a 
CM subfield Lq, and hence also its totally real subfield F. Either F = Q, 
in which case Lq is imaginary quadratic, or F 7^ Q. Either way there 
will be a proper subfield M of degree < 3, and so the purported equality 
L(s, 11(g)) = L(s,X) cannot happen. We are now done with the proof of 
Theorem IU1 

Proof of Theorem [3 By assumption, the £-adic representation p is 
functorially associated to the cuspidal cohomological form 7r(g) on GL(3)/Q 
with q £ T2. 

Lemma 5.1. p is irreducible under restriction to any open subgroup. 

Proof. Suffices to show that the restriction pe to Gal(Q/.E) is irreducible 
for any finite Galois extension E/Q. Pick any such extension and write 
G =Gal(£'/Q). Suppose pe is reducible. Then we have either 

(i) PE — t © x with r irreducible of dimension 2 and x °f dimension 1; 
or 

(ii) pe — Xi © X2 © X3> with each Xj one-dimensional. 

Let V be the 3-dimensional Q^-vector space on which Gal(Q/Q) acts via 
p. Suppose we are in case(i), so that there is a line L in V preserved by 
Ga^Q/E 1 ) and acted upon by x- Note that G acts on {r, x} and, by the 
dimension consideration, it must preserve {x}- Hence the line L is preserved 
by Gal(Q/Q), which contradicts the fact that p is irreducible. 

So we may assume that we are in case (ii). We claim that Xi / Xj if i 3- 
Indeed, since p arises as (the base change to of) a summand of the £-adic 
cohomology of a smooth projective variety, it is Hodge- Tate, and so is each 
Xj- So each Xj 1S locally algebraic and corresponds to an algebraic Hecke 
character x'j of E. By the identity of the L-functions, we will have L s (s, it) = 
Y[j L s (s, x'j) for a suitable finite set 5 of places 5. By the regularity of n, 
each x'j must appear with multiplicity one, which proves the claim. Now let 
Lj denote, for each j < 3, the (unique) line in V stable under Gal(Q/-E) 
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and acted upon by Xi- And G acts by permutations on the set {xi, X2, Xz}- 
In other words, there is a representation r : G — > S3 such that the G-action 
is via r. Put H = Ker(r), with corresponding intermediate field M. Then 
each Lj is stable under Gal(Q/M), so that pu — v\ © ^2 © ^3, where each 
Vj is a character of Gal(Q/M). Also, M/Q is Galois with Gal(M/Q) C S3. 
But from the proof of Theorem ICl that the base change %m of vr to any such 
M is cuspidal. However, if denotes the algebraic Hecke character of M 

defined by i/j, the twisted L-function L s (s,wm © ^j -1 ) wm have a pole at 
s = 1, leading to a contradiction. We have now proved Lemma 5.1. 

□ 

Note that Lemma 1 implies in particular that for any finite extension 
F/Q, Pf does not admit any self- twist. 

Lemma 5.2. For any finite extension E/Q, the restriction pe is not essen- 
tially self-dual. 

Proof. Again we may assume that E/<Q is Galois with group G. As 
before let V denote the space of p, and suppose that we have an isomorphism 
p ~ p v u, for a character v. Then there is a line L in V <S> V on which 
Gal(Q/.B) acts via v. By Schur's lemma (and this is why we have to work 
over Qe), the trivial representation appears with multiplicity one in V (g> V v . 
It implies that v must appear with multiplicity one in V <S> V . We claim that 
V (8> V contains no other character. Indeed, if we have another character u', 
we would have p ~ p©/i, where \i = vjv'. But as noted above, pe admits no 
self- twist, and so p, = 1, and the claim is proved. Consequently, the action 
of G on V (&V must preserve v. In other words, the line L is stable under all 
of Gal(Q/Q), contradicting the fact that p is not essentially self-dual. Done 

□ 

Now consider R = /?©/?'. We know that both p and p' remain irreducible 
upon restriction to any open subgroup and moreover, such a restriction of p 
is not essentially self-dual. It then follows easily that the restriction of R is 
irreducible. 

This finishes the proof of Theorem [Dl 

□ 

6. Proof of Theorem A, Part #1 

By twisting we may assume that ir, it' are unitary, so that 7r v ~ W and 
7r /V ~ it', with respective central characters u,u'. 

Now we proceed in several steps. Applying Langlands's classification, 
C |Lal979-lj . |Lal979-2j . [JS19B11 ). we see that the Kim-Shahidi product 
LT = 7r M 7r' must be an isobaric sum of cusp forms whose degrees add up to 
6. Thanks to the Clebsch-Gordon decomposition 

sym 2 (vr') B n' ~ sym 3 (7r') ffl (vr' ® u'), 
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II is not cuspidal if n is a twist of sym 2 (7r'). 

The list of all the cases when II is not cuspidal is the following: 

Case I: II has a constituent of degree 1, i.e., II = A EE II' for some idele 
class character A and some automorphic representation II' of GL(5). 

Case II: II has a constituent of degree 2, i.e., II = r EE II' for some cusp 
form t on GL(2) and some automorphic representation II' of GL(4). 

Case III: II is an isobaric sum of two cusp forms o\ and a% on GL(3). 

We first deal with Cases I and II. We need some preliminaries. First comes 
the following basic result due to H. Jacquet and J. A. Shalika ( JS1981 , 
|.IS1990j . and R. Langlands f |La!979-l| . |Lal979-2j 1. 

Lemma 6.1. (i) LetH, r be isobaric automorphic representations ofGL n (Ap), 
GLm^Ap) respectively. Assume that r is cuspidal. Then the order of the 
pole of L(s,II®r) at s = 1 is the same as the multiplicity of r occurring in 
the isobaric sum decomposition ofH. 

(ii) L(s, II x II) has a pole at s = 1 of order m = mf ifll = SiUiiTTi 
is the isobaric decomposition of IT, with the 7Tj being inequivalent cuspidal 
representations of smaller degree. In particular, m = 1 if and only if II is 
cuspidal. 

An L-function L(s) is said to be nice if it converges on some right half 
plane, admits an Euler product of some degree m, say, and extends to a 
meromorphic function of finite order with no pole outside s = 1, together 
with a functional equation related to another L-function L v (s) given by 

L(s) = W(d^N) 1/2 - s L v (l -s), 

where IF is a non-zero scalar. 

If vri,7T2 are automorphic forms on GL(m),GL(n) respectively, then the 
Rankin-Selberg L-function L(s,7ri x 112) is known to be nice ( JPSS1983 , 
MW1995 , Shl988 ). Of course, the product of two nice L-functions is 
nice. Furthermore, we recall the following Tchebotarev-like result for nice 
L-functions r |,ISi99()j 1: 

Lemma 6.2. Let Li(s) = Y\ v Li^ v (s) and L2(s) = Y\ v L2, v (s) be two L- 
functions with Euler products, and suppose that they are both of exactly one 
of the following types: 

(a) Lj(s) is an Artin L-function of some Galois extension; 

(b) Lj(s) is attached to an isobaric automorphic representation; 

(c) Lj(s) is a Rankin-Selberg L-function of two isobaric automorphic rep- 
resentations. 

If Li ;1 ,(s) = L2 )V (s) for all but finite places v of F, then L\(s) = L2(s). 
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Proof of Theorem A for Cases I and II. 

Firstly, Case I can never happen. The reason is the following: If A is a 
constituent of II = 7rKl7r', then L(s, II x A) has a pole at s = 1 (Lemma 16. 1JI . 
hence so does 

L(s, tt' x tt ® A) = L(s, tt IE tt' <g> A). 

However, n' and 7r® A are cuspidal of different degrees, hence L(s, tt' x 7r® A) 
is entire, and we get the desired contradiction. 

Now we treat Case II, where II has a constituent r of degree 2. We 
will show that this can happen IF AND ONLY IF tt is twist equivalent to 
sym 2 (7r') in which case r is twist equivalent to tt' . 

In fact, for each finite v where tt and tt' are unramified, 

L v (s, II ® f) = L v (s, tt x (tt' M f )), 

where tt'Mt is the functorial product of tt' and f whose modularity (in GL(4)) 
was established in Ra2()()(ij . One may check the following: If tt' v = a Vt \Sa Vj 2, 
t^v = Pv,i EH Pv,2 EB P Vj 3, and r' v = 7^1 £37^,2, then both sides of the equality is 
the same as \\ { • k L(s, otv,iflv,jlv,k) where the product is over all k such 
that 1 < i, k < 2 'and 1 <j < 3. 
Hence by Lemma lfi.21 

L(s, n (g> f) = L(s, vr x (tt' M f )) 

As r is a constituent of n, the L-functions on both sides above have a 
pole at s = 1. As 7T is cuspidal, this means by Lemma 16.11 tt is a constituent 
of 7r' Kl f. Hence 7r' Kl r should possess a constituent of degree 1, namely a 
character \i. 

Thus L(s, 7r' x r <8> /2) = L(s, 7r' Kl r % Ji) has a pole at s = 1, implying that 
7r' is equivalent to r (g> /i. Hence 

tt' B f ^ /i EB Ad(r) (8) ^, 

which means that 7r = yld(r) (8) /i = Ad^ir') <g> [x. 

Finally, it is clear that if Case II happens, then tt' cannot be dihedral. 
Furthermore, n is Eisensteinian of type (2, 2, 2) if tt' is tetrahedral, and (2, 4) 
otherwise, we can see this by observing that 

tt' M Ad(ir') = sym 3 (vr') (g> oj n > EB tt' (g> w 2 , 



□ 
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7. Proof of Theorem A, Part #2 

It remains to treat Case III. Here again, tt' denotes a cusp form on GL(2) 
and vr a cusp form on GL{3). Assume that II = a\ EE a<i where o\ and o-i 
are cusp forms on GL(3). 

We will divide Case III into two subcases: In this section, we will assume 
that tt' is not dihedral. The (sub)case when tt' is dihedral will be treated in 
the next section. 

The following equality is crucial, and it holds for all cusp forms vr' on 
GL(2) and vr on GL(3): 

Proposition 7.1. 

L(s,tt x vr'; A 3 <g> u~ x)L{s, tt' (g) uj n >x) 

(1) = L(s, sym 3 (7r') ® %)L(s, (vr M tt') x tt (g> u w >x) 
where u n i and uj n are the respective central characters of tt' and tt. 

Proof of Proposition \7. 1\ 

We claim that both sides of (^Q) are nice. Indeed, we see that formally, 
the admissible representation A 3 (vr M tt') (g) is equivalent to sym 3 (vr') EE 
{Ad{Tr) M tt' <S> uv). So the left hand side is nice. And the right hand side 
is nice by [KSh2002-l , whence the claim. So by Lemma 16.21 it suffices to 
prove this equality given by the Proposition locally at v for almost all v. It 
then suffices to prove the following identity (as admissible representations) 
for almost all v. 

A^vr^vr^u^EBvr;®^, 

(2) = sym 3 «) EE vr; Mtt v Mn v ® w ff # 

Let v be any place where vr' and vr are unramified. Say vr^ = a v> \ EE a Vj 2, 
tt v = (3 V) i EE /3„ )2 EE /3„ j3 . Note that u n > v = a V) ia Vj2 and u 7Tv = /3 v ,ip Vt 2Pv,3- 
Then it is routine to check that the left and the right hand sides of (J2J are 
equal to the sum of the following terms: 

Terms A (2 terms): a 3 1 EE a 3 2 ; 

Terms B (2 x 4 = 8 terms): 4 copies of (a Vj i EE a v ^) ® ^ ; 
Terms C (12 terms): ffli<i<2,i<jyK3 

In fact, Terms A, B and C are obtained by expanding the right hand side 
of ©. Since 

Ad(TT v ) = 3 • 1 EE (ffli<#jfe<3, 

the Terms C and (three of) the Terms B are obtained from tt' v M tt v M tt v <g> u^i , 
and the Terms A and (one of) the Terms B arise from sym 3 (vr'). 

The left hand side is easy to handle since we have the following: 
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A 3 (7r(, <g>TT v ) = ^l<i<2,l<j,k<3,jjtkQ. Vj iUJ 7r ' u PyjP Vj k 

EB Zu) nv a v> iu^i v ffl Suj^ay^uJ^ 

In fact, the w" 1 twist of the thing above contributes the Terms C, Terms 
A and (three of) Terms B. 

So we have proved @ , and hence © . 
□ 

Let <7i and 02 be cusp forms on GL(3). 

Lemma 7.2. Let 771 and 7/2 6e i/ie central characters of o\ and 0% respec- 
tively. Then 

L(s, at EB a 2 ; A 3 x') = L{s, r, lX r )L{s, r? 2 x') 
(3) (Ji x ff 2 ® i]2x')L(s, cr 2 x cti <g> r/ix') 

Proof, of Lemma \7.fy 

This is easy since at each place t> where the o~i are unramified, 
A 3 (ct m EB a 2 , v ) = E5 <i<3 (A>i,„) B A 3 "*^,,,)) , 

A 2 (a i> „) = cTi,„ <g> r/i 

and 

A 3 ^) =r/i. 

Done by applying Lemma 16.21 
□ 

Before we apply Proposition 17. II and Lemma 17.21 let us first investigate a 
special instance of Case III when a\ and a 2 are both twists of n: 

Lemma 7.3. If it M tt' = (n <g> Xi) EB (tt EB X2) toen 

sym 3 (7r0 (V <g> uv) EB X ? EB xl 

Hence if tt' is not dihedral or tetrahedral, this cannot happen. 

Proof of Lemma \7.c\ 

Let v be any place where w' v and ir v are unramified. Write 
L v (s,7r') = (1 - U v (Nv)- s )~\l - VviNv)- 8 )- 1 

and 

L v {s,tt) = (1 - - B v {Nv)- s y\l - c v (Nv)- s y\ 
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Then 

L v (s, 7T X 71") = 

(1 - U v A v (Nv)- s y\l - U v B v (Nv)- s y\l - u v c v {Nv)- s y l 

(1 - V v A v {Nv)- s T\l - V v B v {Nv)- s )'\l - V v C v {Nv)- s Y l 

Let X v = xi(Frob^) and Y v = X2(Frob„). Then 
L v (s, tt ® xi)L v (s, IT tg> X2) = 

(1 - X v A v {Nv)- s y\l - X v B v {Nvy s y\l - XvC^NvyT 1 

(i - Y v A v (N V y s y\i - Y v B v (Nvy s y\i - Y v c v (N V y s y l 

Consequently we have 

{ U V A V , U V B V , U V C V , Y V A V , V V B V , V V C V } = 
(4) { X v A v 5 X v B v , X v C v , Yfj A v , Y v B v , Y v C v j- . 

In particular, 

(Etf + C)« + + O = W + + K + C?) 

for each positive integer n. And besides, taking the products of the elements 
on each side of @ and equating, we get 

7t3t/3 _ v3y3 

V V v v 

Now we apply the following lemma. 

Lemma 7.4. If X, Y, U, V, A, B, C are nonzero complex numbers such that 
(for alln> 0) 

(U n + V n )(A n + B n + C n ) = (X n + Y n )(A n + B n + C n ), 
and U 3 V 3 = X 3 Y 3 , then { U 3 , V 3 } = { X 3 , Y 3 } . 
Proof of Lemma \l .J\ 

If A 3 + B 3 + C^O, then U 3 + V 3 = X 3 + Y 3 . Hence { U 3 , V 3 } = 
{X 3 ,Y 3 } as U 3 V 3 = X 3 Y 3 . 

If A 3 + B 3 + C 3 = 0, we claim that A + B + C ^ 0. Otherwise, 

-3ABC = 3AB(A + B) = (A + B) 3 - {A 3 + B 3 ) = -C 3 + C 3 = 

Thus a, b or c is zero. This leads to a contradiction. 

In fact we will prove the following statement: 

Claim: // a,b,c ^ 0, then a + b + c or a 3 + b 3 + c 3 is no£ zero. 

So we claim also that A 9 + B 9 + C 9 + 0, and, A 2 + B 2 + C 2 or A 6 + £ 6 + C 6 
is not zero. 
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Hence 

U n + V n = X n + Y n 
holds for n = 1 and 9, and for one of 2 or 6. 
If this equality holds for n = 1, and 2, then 

{U + V) 2_ {U + V 2 ) {X + Y f_ {X+Y 2 ) 

UV XY, 

implying that {U, V} = {X,Y }, and the lemma will follow. 

Now assume that U n + V n = X n + Y n holds for n = 1, 6 or 9. As we 
have already assumed that U 3 V 3 = X 3 Y 3 , JJ 3n V 3n = X 3n Y 3n . So we have 

{U n ,V n } = {X n ,Y n } 

for n = 6 and 9. 

Without loss of generality, assume that U 9 = X 9 and V 9 = Y 9 . If 
U 6 = X 6 and V 6 = Y 6 , then of course we have U 3 = X 3 and V 3 = Y 3 
and the lemma follows. If U 6 = Y 6 and V 6 = X G , then U, V, X and Y 
have the same norm. However, since U + V = X + Y, the pairs {U, V} and 
{X, Y} are the same, hence implying the lemma. The reason for this comes 
from the following statement which is elementary: (Note that even when 
U + V = X + Y = 0, although we cannot directly apply this statement, we 
still have U 3 + V 3 = X 3 + Y 3 = 0, so that { U 3 , V 3 } = { X 3 , Y 3 }.) 

Statement The pair (^1,22) such that \z±\ = \z%\ = R and Z1+Z2 = Z is 
uniquely determined by R > and Z with < \Z\ < 2R. 

So in all cases, { U 3 , V 3 } = { X 3 , Y 3 }. 
□ 



Proof of Lemma \773\ (contd.) 
By the previous lemma, 



{U 3 ,V 3 } = {X 3 ,Y 3 } 

at any unramified finite place v. 
Hence 

L(s,sym 3 «)) 

= (1 - U 3 (Nvr s )-\l ~ VfiNv)-')- 1 
(1 - VfU v (Nv)-)-\l - uZVviNv)-')- 1 

= (1 - X*(Nv)-')-\l - Y^Nv)-*)- 1 

(1 - V v u 7r r v (FToh v )(Nvy s y 1 (1 - UvU^FrobvXNv)' 

= H s , Xl,v) L ( s , X2,v) L ( s i n ' ® w <) 
Here we have used 

U V V V = u;^(Prob„) 
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Hence 

sym 3 «) ~ xi, v ffl Xl, v ffl « ® 
and by Lemma 16.21 we get what we desire, namely, 

symV) ~ xifflxi^^®^)- 

□ 

Lemma 7.5. /f 7T ; is tetrahedral with sym 2 (7r') invariant under twisting by 
a cubic character x, then 

symV) ^ (7r / ®w ff /x)ffl(7r / ®w w /x" 1 )- 

Hence the situation of Lemma 17.31 will not happen if 7r' is tetrahedral. 
Proof of Lemma |7.5| 

Consider n' M sym 2 (7r') = sym 3 (7r') ffl (71 7 Oo;,,-/), which obviously contains 
7r' (8) w^/ as an isobaric constituent . 

Since sym 2 (7r') allows self twists by x an d X -1 ! the isobaric sum above 
should also contain ir' <8>uvx an d it' (8>c<VX • Together with n' ® ov, they 
are pairwise inequivalent, the reason being that if a cusp form on GL{2) 
admits a self twist by a character, then such character has to be trivial or 
quadratic. 

Thus, by the uniqueness of the isobaric decomposition, sym 3 (7r') should 
have it' <8>ovx an d vr' (giuvx -1 as hs constituents, and there is no room for 
any other constituent. 

□ 

Proof of Case III when ir' is not dihedral. 

Assume that n = ix M tt' = o~\ EE 02 where o~\ and 02 are cusp forms on 
GL(3) with central characters n\ and 772 respectively. Also, assume that it 1 
is not dihedral. 

Subcase A: tt does not allow a self twist by a nontrivial character. 
From (^Q) (Proposition I7.l)) and (J3J ( Lemma 17.2(1 . we have 

L(s, sym 3 (vr') ® x)L(s, (tt M tt') x tt ® lo^x) 

= L(s, tt' x 7r; A 3 (g) uj^ 1 x)L(s, ir' uvx) 
= L(s, riiu~ l x)L{s, ri 2 u~ l x)L{s, tt' ® <*VX) 
(5) L(s, a\ x a 2 ® ^o^x)!^, cr 2 x 01 (g) ^iw,,: 1 ^) 

Hence, take x = u^n^ 1 , then the right hand side has a pole at s = 1 (as 
the remaining factors do not vanish st s = 1). Then the left hand side also 
has a pole at s = 1. 
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However, since tt' is not dihedral, then sym 3 (-7r') is either cuspidal or an 
isobaric sum of two cusp forms on GL{2) (Lemma 17. 5JI . so any twisted L— 
function of sym 3 (7r') has to be entire. So the only pole at s = 1 should come 
from L(s, (ttMtt') x tt (g) uj^uj^ri^ 1 ). 

As tt is cuspidal, LT = tt tt' should have both a[ = tt <S> oj^ 1 ^ as 
constituents. 

Since tt is not monomial, it does not allow a self twist. Hence, if r/i ^ r/ 2 
then a[ are different. Hence cr^ and a' 2 are the only constituents of n which 
are also twists of tt. Furthermore, if r]\ = rj2, then the order of the pole of 
both sides of ©, and hence also of L(s, (ttMtt') x tt 0uj w iuj w ri^ ), is 2. Hence 
a[ = (7 2 should be a constituent of n with multiplicity 2. 

Thus we get an isobaric decomposition of II as a sum of two twists of 
tt. Thus, from lemma 17.31 and Lemma 17.51 this cannot happen if tt' is not 
dihedral. This completes the treatment of Subcase A. 

Subcase B: tt admits a self twist by a nontrivial cubic character x- 

In this subcase, recall that we are assuming n = tt M tt' = a\ EE 02, where 
u\ and o"2 are cusp forms with respective central characters r}\ and 772- 

We claim that u\ and 02 are also invariant when twisted by x- Otherwise 
&i <8> X and cr, ® x _1 will be distinct from <Tj, while they should both be 
constituents ofn~n®x~n. Hence n has at least degree 3x3 = 9, 
which is impossible as it is automorphic on GL(6). 

Moreover, let v be any place where tt and tt' are unramified. Write (for 
i = 1, 2) tt' v = a v EB a' v , tt v = f3 v EB /^Xu EB /^x^ 1 (this form being implied by 
tt = tt <g> x), and cr^ = 0^ EB ^,uXi; EB Oi^Xv 1 - 

Then we have 

n„ = (a v BE! O (8 a, ® (1 ffl x« ffl Xv 1 ) 

and 

a M BB cx 2 ,„ = (0i,„ EB 2>v ) ® (1 BB x^ EB x" 1 ) 

Since the sets of all cubes of characters occurring in the previous two 
isobaric decompositions should be the same, and since /3 3 = u Wv and 3 = 
rji )V , we must have 

I a^^a'^u^ } = { a 3 /? 3 , a' v 3 [3% j 

= { S l,v^2,v } 

= {m,v,V2,v} 

So 

sym 3 (7r^,) ® = a 3 ^ EB c^ 3 ^ EB (a„ EB a4) IE cj^w^ 
~ J7i )t) EB rj2, v Stt' v ® u^u^, 
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that is 

sym 3 «) = T]i, v u~l ffl m,v^l ffl K ® w^/ 

Thus by Lemma 16.21 or the strong multiplicity one theorem, we have 

sym 3 (vr') ^ r/i^ 1 S ^w" 1 EE n' ® uv 

However, since 7r' is not dihedral, sym 3 (7r') is by Lemma f7.5l either cuspidal 
or an isobar ic sum of two cusp forms on GL{2). This gives a contradiction. 

This completes the treatment of Subcase B. 

□ 

8. Proof of Theorem A, Part #3 

In this part, we will treat the case when tt' is dihedral. After that, we will 
analyze precisely the cuspidality criterion when tt is an adjoint of a form on 
GL(2). Again, F denotes a number field. 

In fact, we will prove the following: 

Theorem 8.1. Let tt',tt be cusp forms on GL(2) / F, GL(3) / F respectively, 
with tt' dihedral. 

Then II = ttMtt' is cuspidal unless both the following two conditions hold: 

(a) There is a non-normal cubic extension K' of F such that ir' K , is Eisen- 
steinian; equivalently, tt' is dihedral of type Dq . 

(b) tt is monomial and tt = I^-,(x') for some idele class character x' of 

K. 

If (a) and (b) both hold, then li is an isobaric sum of two cuspidal rep- 
resentations of degree 3, which are both twist equivalent to tt. 

Before we prove this theorem, let us recall that a dihedral Galois represen- 
tation p' of Qf is said to be of type Dm if its projective image is Dm- It is 
clear that p' is not irreducible if and only if n = 1 (Note that the projective 
image of D^ n must be a quotient of Z?2n since D^ n has a nontrivial center). 
If 6|n, then £>2 n has a unique cyclic subgroup with quotient isomorphic to 
Dq = S3. Suppose K' is a non- normal cubic extension of F, and p' restricted 
to Gk' is reducible. Then the projective image of Qk> should be a subgroup 
of that of Qf of index 3, hence is isomorphic to D 2n /3- Thus n = 3, and p 
must be dihedral of type Dq. Similarly, we conclude that if it' is dihedral 
and ttk 1 is not cuspidal, then tt' is of type Dq. 

Proof of Theorem \H.l\ 

First assume (a) and (b). Note that tt' k , = v\ EE v%, and tt = I^,(x')- 
Then 

n = vr^vr' ^ ir'M i£,(xO 

= Ik'Wk> ® X') = Ik'M) EE l£,(v 2 x') 
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Hence II is not cuspidal. 

Note that tt{ = I^,(vix') MUST be cuspidal as from Section 5, II cannot 
have a character as its constituent. 

Next, we prove that if II is not cuspidal, then (a), (b) and the remaining 
statements of the theorem hold. 

Step 1: ttx is cyclic cubic monomial. 

Assume that n' = I k (t) where r is some idele class character of Ck with 
K a quadratic extension of F. and also assume that K/F is cut out by 5. 

From Section 5, Case I and II cannot happen, so we are in Case III. Say 
II = o"i EB 02 where o~i are some cusp forms on GL(3)/F. As it' allows a 
self twist by S, so does H = ttMtt'. Thus o\ = C2 ® 5 as the only possible 
characters that either o~i allows (for self-twisting) should be trivial or cubic. 

Let 9 be the nontrivial Galois conjugation of K/F. Then ir' K = r EB r e . 
Hence the base change = tt' k Mtik is equivalent to ttk^t plus ttk®{jo9). 
As n = (Ti EB oi <S> 5 hence 11^ is equivalent to the isobaric sum of two copies 
of ai K . 

Thus ttk <8> t = ttr 8> (r o 0) = o~\. As ir' = Ix( T ) i s cuspidal, r/rofl. 
Hence ttk is forced to be cyclic monomial. 

Step 2: it is non-normal cubic monomial. 

By Step 1, fi = r _1 (r o 9) is a cubic character of Ck- Let M be the cubic 
field extension of K cut out by [i. As /x o 9 = M e = M, thus M/F is 
normal, and Gsl(M/F) ^ S 3 . 

Furthermore, ttk = I-^W ^ or some character A of Cm- And also, ttm is 
of the form A EB A' EB A". 

Let K' be a non- normal cubic extension of F contained in M. Then 
[M : K'\ = 2 and ttm is a quadratic base change of ttk 1 - 

We claim that ttx 1 is Eisensteinian, i.e., not cuspidal. The reason is that, 
if ttk 1 were cuspidal, then its quadratic base change ttm would be either 
cuspidal or the isobaric sum of two cusp forms of the same degree. Since 
ttj^i is a cusp form on GL(3), we see from jAC1989j that this is impossible. 

So ttx' must admit a character as an isobaric constituent, which means 
that 7r is induced from some character of Ck 1 - 

Step 3: tt' k , is not cuspidal, hence tt' is dihedral of type Dq. 

Recall that ir' K = tEB(to#) = tEB r/i, so that ir' M = tm EB tm as M/K 
is cut out by \x. 

Thus the projective image of p' M is trivial, where p' is the representation 
Ind^(r) of the Weil group Wf, and p' M is the restriction of p to Qm- Hence 
the projective image of p must be Dq. 

Remark: Even if n' is selfdual, r may be a character of order 3 or 6. 
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Step 4-' °~i and 02 are all twist equivalent to n. 
Observe that 

(ri_r 1 )o9 = {t o 9){fi o ey 1 

So r/i -1 is a base change of some character, say of Cf to K/F. 

So tt' = &> v and tt' k , = vk< EB vk'^K'- By Step 2, we may assume 

that 7T = l£,(A) for a character A of a non-normal cubic extension K' of F. 
We get 

U = ttMtt' = tt' 

= Ik'{k' k , ® A) = I^,(uk'X EB vk'\o~k') 
= /f,(A)®i/ffl/|,(A)®^ 
(6) = (vr (8) 1/) EB (vr ® z^) 

Now the proof of Theorem 8.1 is completed. 
□ 

Remark. When 7r is twist equivalent to AoI^ttq), and it' is dihedral, we 
claim that the only way IT = tt M it' can be not cuspidal is for Ad(ir) to be 
non-normal cubic monomial, implying that tt is of octahedral type. We get 
the following theorem which is more precise than the result in |Wa2()()3| : 

Theorem 8.2. Let tt' , and tt" be two cusp form on GL(2)/F, and suppose 
that tt" is not dihedral, then II = tt' M Ad{ir") is cuspidal unless one of the 
following happens: 

(d) tt' and tt" are twist equivalent. 

(e) tt' and tt" are octahedral attached with the same S^-extension, and 
AcKjt') and Ad^n") are twist equivalent. 

(f) tt" is octahedral, tt' = I^(fi) <8> v is dihedral, where fx is the cubic 
character which is allowed by Ad^'^). 

The proof of the theorem above uses the following proposition. 

Proposition 8.3. Letni andiT2 are two non-dihedral cusp forms onGL(2)/F, 
and Ad(jTi) and Ad{rr2) are twist equivalent. Then one of the following holds. 

(g) 7Ti and TT2 are twist equivalent (so that their adjoints are equivalent). 

(h) tt± and tti are octahedral attached with the same S 4- extension, and 
Ad(iTi) and Ad(TT2) are twist equivalent by a quadratic characters. 



Proof of Proposition\KM (cf. |R.a.20()0] l 

It is clear that (g) and (h) imply that tt\ and TT2 are twist equivalent. So 
it suffices to show the other side. 
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First assume that Ad(iri) and Ad(ir 2 ) are equivalent. 
Consider II = 7Ti Kl tt 2 . Note that 

II Kl IT = 1 EE Ad(ni) EH Ad(ir 2 ) EH Af(TTi) Kl Ad(7r 2 ) 

admits two copies 1 as its constituents. Hence II is not cuspidal. 

If II contains a character v, then 7F2 = tt\ tg) v. If II is an isobaric sum of 
two cusp forms o\ and a 2 on GL(2), then check that A 2 (7riKl7r 2 ) is equivalent 
to 

{Ad{ir\) E5 Ad^)) 8) LJ ni LJ n2 
which does not contain any character constituent; However, A 2 (di EH 172) is 
equivalent 

to ai muj ai m {ax Mo 2 ) 

which contains two GL(l)-constituents. Thus we get a contradiction, and 
(g) holds. 

Furthermore, assume that 

Ad(ir 2 ) = Ad(iri) (g) e 

where e is a character. Then 

Ad(n 2 ) = Ad{iri) <g> e" 1 

and hence 

Ad(ni) m Ad{-K{) = Ad{n 2 ) M Ad(ir 2 ). 

However, 

Ad(TTi) M Ad(TTi) ^ 1 EH Ad(iri) EH A\m). 

Hence 

Ad(iri) EH ^ 4 (vri) Adfo) EH ^ 4 (vr 2 ). 

If Ad(iri) and A<i(7r2) are equivalent, we get (g). Otherwise, Ad{n\), which 
is a nontrivial twist of Ad{ir 2 ), must be contained in A 4 (^2). 

So 7Ti and 7T2 are of solvable polyhedral type. 
If 7T2 is tetrahedral, then 

A 4 (tt 2 ) = Ad(ir 2 ) EH w EBw 2 

where u; is some cubic character that Ad(n 2 ) admits as as self- twist. So this 
cannot happen. 

Thus ir 2 and tti are octahedral, and 

A a {k 2 ) ^ Ad{v 2 ) ® e 

where K is a quadratic extension of F such that Acf^^-) allows a self twist 
by /i, and e is the quadratic character cuts out K. 

So they must come from the same 54-extension, and hence (h) holds. 
□ 

Proof of Theorem Iff.M 



A Cuspidality Criterion with a cohomological application 



23 



Set tt = Ad(ir") From what we have seen (including the proof of Theo- 
rem l8.1|l . II = tt' M AcI(tt') is cuspidal unless (i) Ad(jr') and Ad(n") are twist 
equivalent; or (ii) tt' is dihedral of type Dq, tt = Ad(Tr") is non-normal cubic 
monomial, and (a) and (b) of Theorem IS.ll hold. 

If (ii) holds, then tt" must be octahedral, and (f) must hold. If (1) holds, 
then Ad(jT') and Ad^n") are twist equivalent. Then part (g) of Proposi- 
tion |H31 implies (d), and part (h) of this proposition implies (e). 

□ 

The proof of Theorem A is now complete. 

9. Proof of Theorem B 

In this part we deduce Theorem IBl from Theorem lAl First we need some 
preliminaries. 

Recall that a cusp form tt on GL{n) over F is essentially selfdual if tt is 
twist equivalent to tt. Throughout this section, tt' and tt denote cusp forms 
on GL{2) and GL(3) over F. We assume that tt' is not dihedral, and tt is 
not twist equivalent to Ad(Tr") for any cusp form tt" . 

First, from Theorem lA"l we have the following: 

Corollary 9.1. I/tt' is not of solvable polyhedral type andir is not essentially 
selfdual, then H = tt Kl tt' is cuspidal. 

Lemma 9.2. Let K be any solvable extension of F. If tt is not essentially 
selfdual, and if ttk does not admit any self twist, then ttk is not essentially 
selfdual. 

Proof of Lemma Ifl.M 

First, assume that [K : F] = I a prime so that K/F is cyclic. Let 0^1 
be a Galois conjugation of K over F, and r be a character cutting out K/F. 

Assume that ttk is essentially selfdual. Say ttk — <8> [i, for a character 
\x. Applying 6, and being aware of the fact that ttk is fixed by 6, we get 

t?k = k k ® {fJL o 6) 

Since ttk does not allow a self twist, then \x o = fi, hence \i must be a 
base change of some character a of Cf to K. 

Hence, tt and tt® a have the same base change over K/F, and thus must 
be twist equivalent. This shows that tt is also essentially selfdual. 

In general case, let K$ = F C K\ C ... C K n = K be a tower of 
cyclic extensions of prime degree. Assume that TTx n = t^k is essentially 
selfdual, then as TTK n does not allow a self twist, neither does TTKi for any 
smaller i, thus applying the arguments above inductively, we claim that 71"^ 
is essentially selfdual. In particular, tt must be essentially selfdual. 
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□ 

Proof of Theorem lijl 

First prove (a). II is cuspidal from Corollary 19, 11 First, assume only that 
tt is not essentially selfdual and does not allow a self twist. Assume that 
II allows a self twist by some character v. Without loss of generality, may 
assume that v is of order 2 or 3. Let K/F be cut out by v. Thus ILk = ifciv) 
is Eisensteinian of type (2, 2, 2) or (3, 3). 

However, ttk is cuspidal as tt does not allow a self twist. From Theorem lAl 
(and the remark at the end of Section 5), ILk cannot be of type (3, 3) as ir' K 
is not dihedral, type (2, 2, 2) as tt' k is not tetrahedral (as tt' and hence tt' k 
is not of solvable polyhedral type). Thus Hk must be cuspidal, and hence 
II does not allow a self twist. 

Moreover, assume that tt is not monomial, in particular, tt is not induced 
from a non-normal cubic extension. Want to prove that II is not either. 

Assume that II = I^i{rf) where rj is some cusp form on GL{2) over K' 
which is non-normal cubic over F. Let M be the normal closure of K' over 
F and E be the unique quadratic subextension of M over F. Then tt' e is 
still not of solvable polyhedral type. And tte is not cyclic monomial as tt is 
not monomial. From Lemma l9,2( tte is not essentially selfdual. Thus the 
first part of (a) implies that He does not allow a self twist. Hence Hm is 
still cuspidal. However, Hm — Ik'(v)m = VM ES (t]M ° EH (tjm ° $' 2 ) where 
0' is the character cutting out M/E. We get a contradiction. 

Thus (a) is proved. 

Next prove (b). It suffices to prove that ILk is cuspidal for any solvable 
extension K of F. 

Since tt' is not of solvable polyhedral type, neither is tt' k . As tt is not of 
solvable type, then ttk must be cuspidal. We claim that ttk cannot allow a 
self twist. Otherwise, say ttk — t^k ®v-v must be a cubic character. Let 
K\/K be cut out by u, then ttk 1 should be Eisensteinian. However, K\/F 
is contained in some solvable normal extension. Thus tt is of solvable type. 
Contradiction. 

Hence the claim. From Lemma 19.21 ttk is not essentially selfdual. Corol- 
lary then implies that Hk is cuspidal. Thus n is not of solvable type. 
□ 

Before we finish this section, we would like to prove the following lemma. 

Lemma 9.3. Let tt be a cusp form on GL{2m + 1) over F . Assume that ir 
is regular algebraic at infinity, and F is not totally complex. Then it is not 
monomial. 

Proof of Lemma \9.,H 

Assume that tt = I^,{v) where K' is a field extension of F of degree 
2m + 1, and v is an algebraic character of Ck- As F is not totally complex, 
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neither is K' as [K' : F] is odd. Thus from Weil ( Wel955 ), v must be of 
the form uq\\ ■ \\ k where v$ is a character of finite order. Thus I^,{v) does 
not contain any nontrivial algebraic character at infinity, and hence cannot 
be regular at infinity. □ 

Appendix 

The object here is to justify the statement made in the Introduction that it 
is possible to construct, for n > 2 even (resp. n = 4), non-selfdual, monomial 
(resp. non-monomial, but imprimitive) , cuspidal cohomology classes for 
rcSL(n,Z). 

Theorem E. (a) Fix any integer m > 1. Then there exists a cuspi- 
dal automorphic representation tt of GL(2m, Aq), which is not es- 
sentially self-dual, contributing to the cuspidal cohomology in degree 
m 2 , and admitting a self-twist relative to a character of order 2m. 
In fact, this can be done for any, not necessarily constant, coefficient 
system. 

(b) There exists a cusp form tt on GL{A) /Q contributing to the cuspidal 
cohomology in degree A, which is not essentially self-dual. It admits 
a self-twist relative to a quadratic character, but not relative to any 
quartic character. 

In both cases it will be apparent from the proof below that there are 
infinitely many such examples, and by the discussion in section 3, they give 
rise, for arbitrary coefficient systems V, to non-selfual, cuspidal cohomology 
classes in H*(T,V) for suitable congruence subgroups T C SL(2m,Z) (m > 
2). Moreover there are naturally associated Galois representations, which 
are monomial in case (a), and are imprimitive but non-monomial in case 
(b). 

Proof, (a) For any m > 1, fix a finite-dimensional coefficient system V. 
Then a cuspidal automorphic representation tt of GL(2m, Ap) contributes to 
the cuspidal cohomology with coefficients in V iff it is algebraic with infinity 
type (in the unitary normalization): (cf. C1988 ) 

{(*/|*|)* , (z/\z\) k \ (z/\z\) k \ {z/\z\) k \ . . . , (z/\z\) k ™, (z/\z\) k -}, 

where (k\, k<i, ■ ■ ■ , k m ) is an ordered m-tuple of integers (determined by V) 
satisfying 

k\ > k 2 > ■ ■ ■ > k m . 
In particular, ir^ is regular. If V ~ C, then as seen in Theorem 3.2, kj = 
2(m—j) + 1. Now pick any cyclic, totally real extension F of Q of degree m, 
and a totally imaginary quadratic extension K of F which is normal over 
Q. Let v\, . . . , v m denote the archimedean places of F, and for each j let Wj 
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be a complex place of K above Vj. Choose an algebraic Hecke character x 
of of K such that 

, \ 2(m-j)+l 

(1) X«y(*) = (^1 l^| 2m_1 Vj<m. 

That such a character exists is a consequence of the discussion on page 3 of 
|Wel955] . To elaborate a bit for the sake of the uninitiated, the necessary 
and sufficient condition for the existence of x as above is that there be a 
positive integer M such that the following holds for all units u £ Q* K with 
components Uj at wy. 

(\ Mki 

But the index of the real units Op in 0* K is finite by the Dirichlet unit 
theorem, and hence for a suitable M, u^ 1 is real for all j. The desired 
identity (2) follows. 

Next pick a finite order character v of K and set 

(3) = xi/. 

Let r be the non-trivial automorphism {complex conjugation) of K/F, and 
5 the quadratic character of F attached to K. Then we may, and we will, 
choose v in such a way that 

(4) I'(I'ot) ^ /ioJV x/Q 

for any character /i of Q, which is possible - and this is crucial - because 
[K : Q] > 4 and so F = {x £ K \ x T = x} / Q. Put 

- := J?(*), 

where / denotes automorphic induction (AC 1989 ). Note that tt makes 
sense because K/Q is solvable and normal, allowing automorphic induction 
to be defined. By looking at the infinity type (1) we see that tt is regular 
and algebraic. 

By construction, tToo contributes to cohomology, and tt is cuspidal because 
the infinity type of precludes it from being $ o a for any non-trivial 
automorphism a of K/Q. To elaborate, note first that r\ := is cuspidal 

and algebraic, corresponding to a Hilbert modular newform on GL(2)/i ? 
of the prescribed weight at infinity. Since the automorphic induction is 
compatible with doing it in stages, tt is just Ip(r]), and since F/Q is cyclic, 
it suffices to check that for any automorphism r of F, rj and r/ o r are not 
isomorphic, which is clear from the properties of 

It remains to check that tt is not essentially self-dual, which comes down to 
checking the same for the 2m-dimensional representation p of Wq induced 
by the character \& of Wk- For this we need, by Mackey, to check that 
7^ (^k)^ o~ for any automorphism a of K and any character \i of Q. 
By our choice of the infinity type, this is clear for any a ^ r (and any fi). 
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For a = r, this is the content of (4). So we are done with the proof of part 
(a). 

(b) Let K/Q be an imaginary quadratic field and j3 a non-dihedral cusp 
form of weight 2 over K with Q coefficients, such that no twist of (5 is a 
base change from Q. Here weight 2 signifies the fact that the Langlands 
parameter of (3^ is given by 

<r(A») = {z/\z\,z/\ Z \} ® (zz) 

Here are two explicit (known) examples with these properties: First consider 
the (non-CM) elliptic curve 

Ei : y 2 + xy = x 3 + (3 + ^3)x 2 /2 + (1 + yf^)x/2 

over K\ = Q(\/—3). This was shown to be associated to a cusp form Pi of 
weight 2 and trivial central character on Gh(2) / K\ by R. Taylor ( |Tal 994 ) 
such that ap (E\) = ap(fti) for a set of primes P of density 1. (In fact, 
recent results can be used to show that this equality holds outside a finite 
set of primes P.) 

For the second example, set K2 = Q(i) and Q the prime ideal generated 
by 8+13i. Then there is a corresponding cusp form fa of weight 2, conductor 
Q and trivial central character, as seen on page 344 of the book EGM1998 
by Elstrodt, Grunewald and Mennicke. Its conjugate by the non-trivial 
automorphism 8 of K2 will have conductor Q e and so no twist of P2 can be 
a base change from Q. There is a corresponding elliptic curve 

E 2 : y 2 + iy = x 3 + (1 + i)x 2 + ix 

over K2 of conductor Q, and one knows for many P that ap(E2) = ap(/?2)- 
Next choose an algebraic Hecke character \ of K such that 

Xoo(z) = (z/\z\) 2 \zz\ 2 . 

For example, we can choose x to be the square of a Hecke character asso- 
ciated to a CM elliptic curve. Now consider, for j = 1,2, the automorphic 
induction 

The infinity types chosen imply that either /3j tg) x 1S n °t isomorphic to its 
transform by the non-trivial automorphism of Kj. So ttj is a cusp form on 
GL(4)/Q. It is cohomological, as easily seen by its archimedean parameter. 
That TTj is not essentially self-dual is an immediate consequence of the in- 
finity types of x an d (3. Finally, ttj admits a quadratic self-twist, namely by 
the character of Q associated to Kj, but it admits no quartic self- twist as 
f3j is non-dihedral. We are now done. 

□ 
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